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(ST, Abstract: Us^sUtt^n^^ethanaco^^^s^^^^^ 

digital "sma«" television or ^^^^f^^^^Z^ cottenf that is " something like" .he programming currenUy 
or televised programmmg and is also able to ^l^^^^ 'f^* ^t^V^^^ intent is sensitive to the piesent context of the 
being viewed or any accessible data element m ont«^L -^c personalized supplemental 

p„gn«nming being displayed and is also P^^^^^^^^^^J^J^Z^ including additional 

^.ent is -acoss media," mcamng that u .s °' ^^f^^^^'J to the viewer in a easy-UKUse, consistent, 

information through broadcasted programs as ^^fj^^^^f^^'^;^;iSized infonnation of interest, and the selecUons 
complementaor and uniform way. The viewer selects <>;y^'*^^; ^"J^on to the viewer in a tin«ly and convenient 
S^"^r.Tl?rirn;L'::^^ " i>or3.ive mt^mg database may also be -sed m 

various ways to supplement the viewer's iHoffle. 
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BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to interactive television, and more particularly to 
the integration of interactive television with various data sources. 
5 Description of Related Art 

Interactive TV merges the mass appeal of traditional TV with the rich, 
interactive power of the Internet. Rudiment forms of interactive TV are available from 
various vendors, including WebTV Networks, Inc. of Mountain View, California, which 
provides receivers for accessing MSNBC interactive television. More advanced forms of 

10 interactive TV are being developed. For example, as more fully described in Backer et al., "A 
Framework for Interactive Television Based on Intemet Standards,'' 
http://www.microsoft.com/ atvef^wpaperl .asp, April 14, 1999 (last update date), several 
companies have been collaborating over the past months on a framework for supporting 
interactive television content based largely on the existing Intemet standards. This framework 

15 in fact builds on a number of components, including Intemet Protocol (IP) as the primary 
transport for all data; Session Aimouncement Protocol and Session Description Protocol 
(S AP/SDP) for announcing streams of interactive content; Hypertext Markup Language 
(HTML) and related data formats as mechanisms for describing the presentation of graphics, 
text, video, and other multimedia; Uniform Resource Identifiers (URIs) as the mechanism for 

20 referencing all multimedia elements, including television broadcasts and other content 
delivered over uni-directional links; ECMAScript and the Document Object Model (/.e,, 
JavaScript™ and JScript™) for dynamic control of HTML content and the synchronization 
of HTML content to broadcast video; and Multipurpose Intemet Mail Extensions / Hypertext 
Transport Protocol (MIME/HTTP)-style headers as the representation for content meta-data. 

25 The authors believe that these six elements provide virtually all of the services necessary to 
design and deliver compelling interactive television content, and that only a few elements of 
innovation are required. A specific framework related to this effort is expressed in the 
Advanced Television Enhancement Forum Specification, Draft Version 1.1 r26, Febmary 2, 
1999. 
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Whil= advances are being made in producing and delivering television 
enhancen-enu, fUr*er enhancements are needed U.. use exisflng sunrdards and a pledrora of 
l:lpone:..clmologies.o provide^e Viewer oftelevisionprogranun^gw.*^^^ 
alcessJvarie., of differen.co..n..ypesappropria.e.o*e viewer .naconvemenla^ 

intuitive way. 

— ^'^^'^^en.of^epresen.— „isame.odfordispla.i.con.». 
ftom multiple informadon sources » a view, comprising esublishing a viewer profile for 

the viewer p f u , fXrth^ items of personalized information, the index symbols 

:=ro:::"---ewerf.se.ec«on.d.^^^^ 

0 1 of me index symbols by ^ viewer «. obuin a selected one of flte ,«ms of p^ 
^la^o^andsuperimposing^eselectedi^mof personalized W^^^^ 

■""^ A«,.herembodimen.ofmepresen.invemionisareceiverappa.aU«for 

. .(w,„multinle information sources for viewing by a viewer, compnsmg a 

ttTd:! rpT^-^-dadditionalinformation 
, rr:^^-^^ordrivingaviewingdevice.opresen..eprogram..Heview. 

:^evrerpromeda.abase;animplici.m.ercoupled«,ti.eadaptivev.ew^^ 
^ ^i„,pUc,.fiHerhavi„gani«pu.coupled«>ti.program.rela.edadd,tional 
::^otdelrforgeneratingi.emsofpersonalizedinformationbasedonti,e^^^^^ 
^.ed additional informationando. .he adaptive viewer pn,meda«a,ase. means for 

™ a visual selection aid over the program, the visual selection a,d compns,ng 
" o ri^ofpersonalizedinformationandthei^ 

2iraidbei„gaccessihletoti«viewerfi.se.eotion;means«.identlfymga.^^^^ 

t o 1 index ^bols by the viewer to obtain a selected one of the items of personal 

i:^o:2ransforWi"S----'-----'--"-'"'''' 

30 theprogram. ^^^^^^^^^_„^^^^„.„,^,p_H„vention is a compter program 

^uct comprising a computer readable medium having program logic record»l ti»reon for 
:^rmpul-enab,edapparan..o display content ftommultiplelnformation^ 

Ta vieL o„ a viewing device, comprising- means for establishing a vrewer profile for the 
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viewer based on various interests of the viewer; means for presenting a program to the viewer 
on the viewing device; means for generating items of personalized information based on the 
program and on the viewer profile; means for superimposing a visual selection aid over the 
program, the visual selection aid comprising index symbols for the items of personalized 
5 information, the index symbols on the visual selection aid being accessible to the viewer for 
selection; means for identifying a selection of one of the index symbols by the viewer to 
obtain a selected one of the items of personal infonnation; and means for superimposing the 
selected item of personalized information over the program. 
Brief Description of the Drawings 

10 Figure 1 through Figure 6 are pictorial representations of various viewing 

screen displays over time, in accordance with the present invention. 

Figure 7 is a block schematic diagram of a media fusion system, in accordance 
with the present invention. 

Figiu-e 8 is a flowchart of an illustrative operating session of the media fusion 

1 5 system of Figure 7, in accordance with the present invention. 
Detailed Description of the Preferred Embodiment 

Using little more than a control device such as a handheld infrared remote 
control and a viewing device, which includes, for example, digital "smart" television, a 
standard or high definition television with a set top box, a personal computer, or even a 

20 personal viewing device having a small screen or viewing glasses and a wireless or wired 

Internet connection, a viewer is able to enjoy various pre-recorded or televised programming 
such as movies, sitcoms, theater, musicals, dance, sports, live action, hosted auctions, feature 
advertisements, and so forth, and is also able to view suggestions for supplemental content 
that is "something like" the progranmiing currently being viewed or any accessible data 

25 element in that content. The supplemental content is sensitive to the present context of the 
programming being displayed and is also personalized for the viewer based on the viewer's 
profile. The personalized supplemental content is "across media," meaning that it is obtained 
or derived from various media sources and media types. The viewer may easily select any of 
the personalized supplemental content, which is then presented to the viewer on the viewing 

30 device in a consistent, complementary and uniform way. 

In a standard broadcast television context, for example, the media fusion 
system enhances a viewer's experience by suggesting items of personalized mformation such 
as program information, biographical information on any actors and characters appearing in 
the program, information on the music and effects used in the program, background 
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toformaaon on U>= making of fte program, program re..«d referen.es. historical daU on fte 
program (e.g. movie da«). sports sutistics. interactive menus, advertising infonnaf on. on- 
ineCliing. other statistical or detaiied i„fom«tio„. or a combination thereof that .s re.^ 

both to the program and to the viewer's interesu. The personalized information ts obtamed 
ftom among fte plea».ra of information available from different media sources m,ght mterest 
the viev«r. Tlte media fusion system merges the screen presentation of the suggestions as 
weU as. in some eases, the personalized information itselfwith the regular broadcast 

programming, titeteby pre^ntirtg the infomtation to the viewer in a timely and oonvement 
„,an„errati.erth«. requiring the viewer to access individual piecesofinformationasts 

necessary in a conventional Web-based environment. 

Anillustrative^tampleofaviewer-sexperiencewiththemediafi^onsystem 

is shown in Figures 1 tt^ough 3, which pertain to receiving program related mformation 
during a standard broadcast television program. Figure 1 shows a broadcast pn,gram m 
p^gress after activation of the media teion syst«n. Activation of the media fuston may be 
by manual action of tire viewer or automatically. Manual activation may be in any desHed 
„«nner, as by p^ssingaftmetion button ona^motecontrol or by moving anon-screen 

cursor «. a particular location on ttte screen. Automatic activation may be a pertodtc 
Xationrractivationatpredefinedtimessuchas ti^beginning and end of aprogram and 

daring breaks in the program, for example. ,. , i„ 

A cube 5 appears in ti« upper right comer of the screen (.t may be located m 

any desired place on ti-e screen) «> indicate ti». the media fusion system is active and *a, 
Jle items of personali»d information are made available to tire viewer, as md.c.ed W 
™„bols on three facets of *e cube 5. If desired. U» cube 5 may be programmed to rota.^ 
::„command.providlng«p.osix^.stoindica«ti»avai.abiUt,ofitemso^^^^ 

; ilnnation. Alternatively, any tirree dhnensional shape having a multipltcty of ftc^ts may 
loused. Multiple cubes or oti^ three dimensional shapes may he used if more ftcets are 
re,uired.orolderitemsofpersonalizedinfom.tionm.yheretiredasnewer..«^of 

Zonal^ infomtation become available. Two dimensional matrices and one drntenstona. 

he used as an alternative to ti«e dimension- shapes, if desired. alti»ugh these are 
0 most ulfi. as nested menus to aid ti,e viewer in making interactive choirs — 

Prized infom-ation has been accessed. The symbol appearing on a facet to mdt^u *a. 
L item of personalized infomtation is available may be an arbitiary number such as sho™ m 
ngures 1 lugh 3. or may be an icon such as a musical note, talking head, motion ptctirr. 
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camera, dollar sign, exclamation mark, and so forth that suggest the general nature of the 
item of personalized information referenced by the facet. 

The cube 5 is just one specific type of a visual selection aid. While the visual 
selection aid may be one dimensioned or multi-dimensioned, it basically functions to provide 
the viewer with a convenient visual clue for selecting items of personalized information. The 
items of personalized information are indexed by any suitable symbols, which are displayed 
to the viewer on the visual selection aid. 

Although the viewer may ignore the cube 5 and continue watching the 
television program, typically the viewer desires certain information or wishes to take certain 
actions related to the current television program. This is done by "selecting" a facet of the 
cube 5. How a selection is made depends on the type of controller used by the viewer. For 
example, a selection may be made with a system that supports an on-screen cursor by moving 
the cursor over the desired facet and pressing the select button, and selection may be made 
with a system that does not support an on-screen cursor by pressing a function button and 
then the number corresponding to the desired facet. 

Figure 2 shows the result when the viewer selects the facet labeled "1 The 
item of personalized information represented by facet "1" is program related information 
identified by the title "1. On Safari (documentary)," which appears on the screen. Because 
the viewer of Figure 2 regularly watches wildlife and conservation programming, the media 
fusion system has presented an item of information that pertains to other conservation 
programming. The media fusion system accomplishes this by maintaining an adaptive profile 
of the viewer. The ability of the viewer's adaptive profile to generate relevant and valuable 
recommendations and links improves with experience. 

If the viewer is not interested at the moment in more information on the "On 
Safari" documentary, the viewer may cancel the selection in any convenient manner, such as, 
for example, by pressing a "cancel" button on a remote control or moving a cursor over a 
"cancel" icon (not shown). The number "1" remains on a face of the cube to indicate that the 
item of personalized information is still available if the viewer later desires to access it. The 
nomial advertisement stream and other ancillary content continues to be presented to the 
viewer. However, if the viewer is interested in more information, the viewer makes a further 
selection in any convenient manner, such as, for example, by again selecting the facet "1" or 
by selecting the displayed text. Additional program related information is displayed, 
illustratively "Documenting a team of explorers, we find them fighting for their lives while 
on safari in Kenya .... Catch this program tonight: Discovery Channel: 8 PM" as shown in 
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Figure 3 The viewer may use this additional program related information in any desired 
manner, including, for example, simply remembering to tune in at 8 PM to watch the 
program or by initiating the automatic programming of the viewer's VCR to record the 
program. Since the additional program related information is superimposed over the current 

5 broadcast program, the viewer simply cancels the selection to return to the program when her 
use of the additional program related information is complete. 

Other facets of the cube represent other types of personalized information. An 
illustrative example of a viewer's receiving advertising information with the media fusion 
system during a standard broadcast television program is shown in Figure 4 and Figure 5. As 

10 thewildlifeprogramprogresses.theviewerselectsthefacetofcube51abeled"2.''Fig^ 

shows the result of this selection, which is a display on the screen of «2. Landrover 
(advertisement)." The advertisement is based on the current program content as well as the 
viewer's adaptive profile, so the advertising is more likely to be of specific value to the 
viewer. 

If the viewer is not interested at the moment in more information on the 
"Landrover" advertisement, the viewer may cancel the selection. The number "2" remains on 
a face of the cube to indicate that the item of personalized information is still avaUable if the 
viewer later desires to access it. The normal advertisement stream and other ancillary content 
continues to be presented to the viewer. However, if the viewer is interested in more 
infomiation, the viewer makes a fiirther selection in any convenient mamier, such as, for 
example by again selecting the facet "2" or by selecting the displayed text. The full 
Landrover advertisement appears superimposed over the program, as shown in Figure 5. The 
full advertisement is, illustratively, "The New 4x4, the best in its category, is ready for a new 
type of owner. Ready for an owner like you . . . ." The viewer may use this advertisement m 
25 any desired manner depending on the capabilities of the viewer' s set top box, ranging firom 
simply taking note of it for general interest to automatically contacting an on-line vendor 
based on the displayed information to schedule an appointment for test driving the vehicle or 
even to directly purchase the vehicle. 

An illustrative technique for using, for exaniple, the related program 
30 infomiation shown in Figure 3 or the Landrover advertisement shown in Figure 5 as a 

gateway to other viewer activities is to enable the viewer to evoke a menu or a nested set of 
interactive menus by again selecting the appropriate facet or by selecting the displayed 
advertisement. TTie menu may have a number of choices, including for the related program 
irfomiation example a "Program VCR" choice and for the Landrover advertisement example 
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an e-commerce choice. For some activities, display of the program is discontinued at least 
temporarily so that the activity screen may be displayed. 

The third item of personalized information represented by the third facet of 
cube 5 is, for example, an article on native African wildlife. This is because the viewer 
5 frequently consults on-line reference material while watching nature programming, a 
behavior captured in the viewer's adaptive profile. 

Other examples to illustrate on-line shopping, biographical information, and 
interactive programming follow. 

Example 1: On-line Shopping, A viewer is watching an advertisement for a particular model 
10 of car. The viewer is interested in additional information about the car and "selects" the 

image of the car. The media fusion system responds with information relevant to the viewer's 
interests, such as the retail price of the car in the viewer's geographical region, Web sites or 
dealerships from which the car may be purchased, safety information about the car (the 
media fusion system having "learned" that the viewer is safety-conscious from previous 
1 5 viewing selections), and a video of the car beuig driven on a test track (the media fusion 
system having "learned" that the viewer likes action scenes from previous viewing 
selections). The relevant information is obtained from a variety of sources, both local and 
from the Web, 

Example 2: Program Related References, On-line Shopping, and Enhanced Communications, 
20 The viewer of a movie enjoys motion picture soundtracks, a behavior captured in the 

viewer's adaptive profile along with the viewer's frequent use of e-commerce and email. 

Finding that he or she enjoys the music, the viewer notices that one of the items of 

personalized information made available by the media fusion system is a "current audio" 

selection. The viewer selects this item, and the system responds by showing the composer, 
25 artist and title. The viewer selects "further options," and is presented with a screen menu of 

personaHzed options, including, for example, a music purchase option and an email option. 

These options permit the viewer to effortlessly do a number of actions common to the viewer, 

such as purchasing the music for herself, or informing friends of the music, or purchasing the 

music for a friend and informing him that it is coming. 
30 Example 3: Movie Database, The viewer is watching a movie. As the movie ends, the viewer 

is presented with various items of personalized information, including a suggestion to watch 

another movie similar to the movie just viewed. 

Example 4: Detailed Information. The viewer engages in frequent stock transactions with an 
on-line broker, a behavior captured in the viewer's adaptive profile. During a commercial 
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advertisement for a product from a particular company, the viewer is presented with various 
items of personalized information, including a financial report on the company and a link to 

the viewer's on-line broker. 

An illustrative media fusion system, which includes various computers, local 
area and global networks, enabling software, and databases, is shown in Figure 7. The 
viewer's portion of the media fusion system, the viewer's "receiving equipment," may be 
implement in any of a variety of ways, including contained in a separate housing such as a set 
top box, buih into a broadcast television, or rumiing as an application on a computer or 
another'type of smart appliance having an attached monitor. Broadcast programming and 
Internet access is available by a variety of communications techniques, including radio, 
cable, satellite, and phone line. Set top boxes are presently available from a variety of 
vendors, including, for example. General Instruments, Inc. of Horsham, Pemisylvania, and 
Scientific Atlanta, Inc. of Atlanta, Georgia, and even more powerful set top boxes are m 
development and production. The set top box may be a Java console, for example. Where the 
media fiision system is implemented using the Internet, suitable Internet-resident computers 
include, for example, servers and clients. Suitable computers, communications networks, and 
set top boxes are well known in the art. 

On-screen results 40 (Figure 7) are displayed on a television monitor 
connected to the set top box. It will be appreciated that the use of a television monitor is - 
illustrative, and that many other techniques are well known to persons of ordinary skill in the 
relevant arts for displaying dynamic graphical data. In this example, the on-screen results 40 
include three components, a TV program component 42. a enhanced behavior component 44. 
and a server-based content component 46. These components originate generally fi-om data 
20 It will also be appreciated that various circuits and methods for processing dynamic 
graphical data, including overlaying graphical data from different sources, are well known to 
persons of ordinary skill in the relevant arts. 

The TV program component 42 originates from a primary broadcast 22, and 
typically is carried by FM tmismission. satellite, or cable. The TV program component 42 
may also be received in other ways, such as over DSL or ISDN or other types of advanced 
30 digital communications techniques, and from prerecorded materials such as video tape, 
digital tape. CD-ROMs, and DVDs. Program selection is in accordance with viewer 
selections 50. 

The enhanced behavior component 44, which includes enhanced behavior 
information such as links related to the content of the primary broadcast 22 and other 
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information designed to enhance the behavior of the viewer's receiving equipment, originates 
from additional information 24. The additional information 24 is coordinated with the 
primary broadcast 22 and preferably originates with it, is context sensitive by design, and 
illxistratively accompanies the primary broadcast 22 in the vertical blanking interval (for 
analog television), for example. The additional information 24 is filtered by an 
explicitAmplicit filter 14 and the resuhing enhanced behavior information 44 is superimposed 
over the TV program 42. The explicit/implicit filter 14 also acquires information about the 
televised program context from the additional information 24. 

The server based content component 46 originates from various remote data 
sources such as servers 28 and 29, based on requests, illustratively URL or CGI instructions, 
from the explicit/implicit filter 14. Servers 28 and 29 may reside on a local area network, on 
an enterprise network, or on the Internet, for example. In the event that servers 28 and 29 
reside on the Internet, they are compliant with Web protocols. The Internet connection allows 
many different types of information and extended connections to be provided to the viewer 
while a program is being viewed. Examples of types of information that a viewer may 
requested and that may be displayed as server based content 46 are an article about African 
vsdldlife from the New York Times Web site and detailed advertising information and vehicle 
specifications from the Landrover Web site. An example of a type of extended connection 
supported is a purchase option, by which a viewer can conveniently purchase an advertised 
product or service on-line. 

The explicit/implicit filter 14 provides filtering for information from various 
sources, so that only context-sensitive information that is within the scope of the viewer's 
profile is furnished as the enhanced behavior component 44. The performance of the 
explicit/implicit filter 14 is dependent on the viewer's personal profile, as defined by 
information contained in the adaptive viewer profile database 12, and on the viewer's 
selections 50. The explicit/implicit filter 14 acts on the additional information 24 by blocking 
enhanced behavior information that is not within the scope of the viewer's personal profile 
and passing enhanced behavior information that is within the scope of the viewer's personal 
profile. The explicit/implicit filter 14 also requests information from the servers 28 and 29 as 
required by the viewer's selections 50 and in accordance with the viewer's personal profile. 

Through implicit filtering, the adaptive viewer profile database 12 captures the 
personality of a particular viewer (or several viewers, if more than one viewer uses the set top 
box), and is progressively refined and augmented to create an adaptive profile based on the 
viewer's history together with marketing statistics. The personal profile develops over time. 
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in that the system "leams" what type of infonnation to offer to the viewer based on vanous 
criteria such as what the viewer usually selects and the amount of time that the selection 
holds the viewer's interest, irrelevant, and adapts accordingly. For example, if advertxsmg 
information rather than reference material is usually selected, the system works out a ratto to 
reflect the viewer's behavior and uses the ratio in responding to the viewer's current 
programming choice or specific request. 

Explicit filtering involves building or supplementing a viewer's profile based 
on her answers to specific questions. For example, detecting a new viewer, the 
explicit/implicit filter 14 presents a pre-programmed list of questions to the viewer to 
establish an initial profile, thereby allowing the media fiasion system to function on at least a 
nxdimentary level initially. Explicit filtering is also usefi.1 for confirming trends identified by 
the explicit/implicit filter 14, thereby saving time in updating the viewer's profile, and for 
resolving any ambiguity that may be detected in a viewer's developed profile by posmg 
questions to the viewer to resolve the ambiguity. Implicit filtering involves bmldmg a 
viewer's profile based on her selections, including television program selections, items of 
personalized information accessed, Web links accessed, Web sites explored, purchases made, 
and so forth. The personal profile of each viewer using the media fusion system is stored m 
an adaptive viewer profile database 12, which preferably resides on the set top box along 
with the explicit/implicit filtef 14 and is locally accessible to the explicit/implicit filter 14.. 

Implicit filtering may, if desired, draw fi:om collaborative data in buildmg a 
viewer's profile. For example. Figure 7 shows that the explicit/implicit filter 14 has a bi- 
directional path to a profile mirror 27, which is associated with a collaborative filtenng 
database 26 and a server 29. The profile mirror 27 may be another computer or another 
process running on the same computer as the server 29. It is not part of the collaborative 
filtering database 26, but rather includes profile information derived fi-om the collaborative 
filtering database 26 and, if desired, from the adaptive viewer profile database 12. The server 
29 illustratively has "push" capabilities, which enable the server 29 to push meamngful 
information to the viewer based on information from the profile mirror 27. 

The collaborative filtering database 26 is a sizeable database of profiles of 
30 viewers who visit the Web sites hosted by the server 29. A number of these viewers are likely 
to have profiles that are similar to the profile of the viewer whose profile is maintained in the 
adaptive viewer profile database 12. These similar profiles are identified by comparing the 
viewer's profile with the collaborative filtering database 26. Differences between the 
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viewer's profile and the similar profiles in the collaborative filtering database 26 represent 
possible viewer interests, and as such are useful to the viewer in a variety of ways. 

One way in which the differences between the viewer's profile and the similar 
profiles in the collaborative filtering database 26 is useful is as part of a response to one of 
5 the viewer selections 50. In this case, the differences are furnished by the profile mirror 27 to 
the server 29 which pushes information to the viewer as server based content 46. 

Another way in which the differences between the viewer's profile and the 
similar profiles in the collaborative filtering database 26 is usefiil is to create additional items 
of personalized information under the assumption that the viewer might be interested in the 
10 same things as other viewers who have similar profiles. In this case, the differences are 
fumished by the profile mirror 27 to the explicit/implicit filter 14 which includes symbols 
that index the differences on the visual selection aid as items of personalized information. If 
the viewer selects any of them, the selection is used to update the adaptive viewer profile 
database 12. 

1 5 Another way in which the differences between the viewer's profile and the 

similar profiles in the collaborative filtering database 26 is useful is to augment the profile of 
the viewer whose profile is maintained in the adaptive viewer profile database 12, imder the 
assimiption that the viewer will be interested in the same things as other viewers who have 
similar profiles. The augmentation is performed in any convenient manner. For example, one 

20 technique involves copying the adaptive viewer profile database 12 to the profile mirror 27, 
updating the profile mirror with appropriate information from the collaborative filtering 
database 26, and writing the updated viewer profile back to the adaptive viewer profile 
database 12. Another technique involves copying the adaptive viewer profile database 12 to 
the profile mirror 27 so that the differences may be identified, then furnishing the differences 

25 to the explicit/implicit filter 14 which handles updating the adaptive viewer profile database 
12. The new data in the viewer's profile is used by the explicit/implicit filter 14 to generate 
items of personalized information, and is evaluated over time by the explicit/implicit filter 14 
just as is other data in the viewer's profile. 

The collaborative filtering database 26 is also useful for establishing a "starter 

30 kit," or an initial profile of data for a new user. Preferably, the new viewer is asked a number 
of explicit start up questions as part of the explicit filtering process by the explicit/implicit 
filter 14, and then the adaptive viewer profile database 12 is augmented from the 
collaborative filtering database 26. 
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Suitable algorithms and software for performing data collection and filtering 
and for maintaining databases such as the adaptive viewer profile database 12 and the 
collaborative filtering database 26 is well known to persons of ordinary skill in the art. 
Generally known as software agents, suitable software is based on any of a variety of 
5 techniques, including Bayesian probability-based recommendation models, decision tr«e 
models, neural network models, and distance metrics. The database itself may be organized 
in any suitable way, including flat file, relational, and object oriented. Examples of the 
Bayesian probability-based recommendation model are described in, for example, the 
following articles, which hereby are incorporated herein in their entirety by reference: 
10 Pazzani, M.. Muramatsu, J., and Billsus. D., SyskiU & Webert: Identifying Interesting Web 
Sites in Proceedings of the National Conference on Artificial hitelligence, Portland, Oregon, 
1996- Billsus, D. and Pazzani, M., Learning Probabilistic User Models, in Workshop Notes 
of Machine Learning for User Modeling, Sixth International Conference on User Modeling. 
Chia Laguna, Sardinia, 1997; and Pazzani, M. and Billsus. D., Learning and Revismg User 
15 Profiles- The Identification of Interesting Web Sites, in Machine Learning 27, 1997. pp. SU- 
SS 1 . An example of collaborative filtering is the technology being researched at the Media 
Laboratory of the Massachusetts Institute of Technology, Cambridge. MA and 
commercialized by such companies as Firefly Networks, Inc. of Cambridge. MA. 

To enhance the seamlessness of media fusion, the primary broadcast 22 may 
20 be provided with embedded highlights. Embedded highlights are associated with specific 
objects on the screen, with the location of those objects being fimiished as part of the 
additional information 24 or in any other suitable mamier. A highlight is of any suitable fomi, 
such as. for example, a colored region or tag that tracks with the movement of its associated 
screen object. For example, the screen object may be a sweater worn by an actor, and the tag 
25 may be a colored rectangle that is superimposed on the sweater and moves with it. The 
highlight may be selected by the viewer in any convenient manner, in which event 
information is fiimished about the item and other options, such as a purchase option, are 
presented. If the tag is not selected, it fades from the screen. 

Figure 8 is a flowchart showing an operating session of the media fiision 
30 system of Figure 7. Figure 8 is merely illustrative, since the media fusion system may be 
programmed in a variety of ways depending on the experiences, skill and preferences of the 
system architect and computer programmers. When the viewer's receiver is turned on (step 
100) the viewer's receiver determines using a viewer ID and password or other such 
mechanism whether the viewer is a new viewer or a previous viewer. If the viewer is new. 
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the receiver (through the explicit/implicit filter 14 of Figure 7, which preferable is resident in 
the receiver) poses various questions to establish a start-up profile for the new viewer (step 
1 12). Once having established a few general traits about the new viewer, the receiver (agam 
through the explicit/implicit fiher 14) may, if desired, augment the new viewer profile from 
5 the remote collaborative filtering database 26. Alternatively, a new viewer may adopt another 
viewer's profile or may choose an initial profile from among a number of pre-constructed 
"standard" profiles. Having created a viewer profile, the receiver then generates and makes 
available for selection new items of personalized information (step 1 14). Depending on the 
results of the decision step 110, either a newly created viewer profile or an existing viewer 

10 profile is used in subsequent operation. 

As additional information 24 becomes available through the primary broadcast 
22 (step 120), the receiver updates the viewer's profile in accordance therewith (step 122) 
and generates and makes available for selection new items of personalized information (step 
124) based on the new additional information and the viewer's updated profile. Alternatively, 

15 the additional information 24 may be stored and the adaptive viewer profile database 12 
updated periodically or when the storage is nearly fiill. 

If the viewer makes a selection (step 1 30), the receiver responds in an 
appropriate manner. For example, if the selection is a channel selection or program change 
(step 140), the receiver changes the channel and updates the viewer's profile accordingly 

20 (step 142), and generates and makes available for selection new items of personalized 

information (step 144) based, for example, on the known characteristics of the new channel 
(news, weather, sports, general entertainment) and the viewer's updated profile. 

If the viewer's selection is of an item of personalized information or an 
embedded tag (step 150), the receiver updates the viewer's profile accordingly (step 152), 

25 and generates and makes available for selection personalized information and/or options (step 
154) based on the viewer's selection and the viewer's updated profile. A personalized option 
may itself have the characteristics of an item of personalized information, in that it may be 
selected in a subsequent execution of step 150 and acted upon in subsequent executions of 
steps 152 and 154. 

30 Personalized options may include such choices as a purchase option, a chat 

option, a comparison shopping option, and email option, a Web site browse option, and so 
forth. If the viewer's selection is of such a personalized option (step 1 60), the receiver 
updates the viewer's profile accordingly (step 162) and executes the option in the appropriate 
manner (step 164). 
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The various methods and apparatus described herein in some aspects enhance 
a viewer's viewing experience by providing easy access to relevant information, and in some 
aspects provide a viewer of television programming with greater access to a variety of 
different content types appropriate to the viewer in a convenient and intuitive way. 

The description and applications as set forth herein are illustrative and are not 
intended to limit the scope of the invention, which is defmed in the following claims. 
Variations and modifications of the embodiments disclosed herein are possible, and practical 
alternatives to and equivalents of the various elements of the embodiments are known to 
those of ordinary skill in the art. These and other variations and modifications of the 
embodiments disclosed herein may be made without departing from the scope and spirit of 
the invention, as set forth in the following claims. 
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1 . A method of displaying content from multiple information sources to a viewer, 
comprising: 

establishing (100, 1 10, 1 12) a viewer profile for the viewer based on various 
interests of the viewer; 
5 presentmg a program (22) to the viewer on a viewing device (40, 42); 

generating (114, 124, 144) items of personalized information based on the 
program and on the viewer profile; 

superimposing a visual selection aid (5) over the program, the visual selection 
aid comprising index symbols (1, 2, 3) for the items of personalized information, the index 
1 0 symbols on the visual selection aid being accessible to the viewer for selection; 

identifying (150) a selection of one of the index symbols by the viewer to 
obtain a selected one of the items of personal information; and 

superimposing the selected item of personalized information (44, 46) over the 

program. 

15 

2. A method as in claim 1 further comprising updating (152) the viewer profile in 
accordance with the viewer's selection of the selected item of personalized information. 

3. A method as in claim 1 fiuther comprising augmenting the viewer profile in 
20 accordance with a collaborative data base (26). 

4. A method as in claim 3 wherein the personalized information generating step 
comprises: 

generating at least one of the items of personalized information (27, 29) based 
25 on user profiles in the collaborative database (26) similar to the viewer profile (12), at least 
one of the index symbols on the visual selection aid being an additional index symbol 
therefor; 

identifying a selection (50) of the index symbol for the item of personalized 
information based on the similar user profiles in the collaborative database; 
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updating the viewer profile (12, 14) in accordance with the viewer's selection 
of the item of personalized information based on the similar user profiles in the collaborative 
database; and 

superimposing the item of personalized information (46) based on the similar 
5 user profiles in the collaborative database over the program. 



5. 

comprises: 



A method as in claim 1 wherein the personalized infonnation generating step 
detecting additional information (24) related to the program; 



filtering the additional information in accordance with tiie viewer profile (14) 

to obtain filtered additional infonnation; and 

generating at least some of tiie items of personalized information (44) from tiie 

filtered additional information, 

^5 6 A method as in claim 1 wherein the personalized infonnation generating step 

comprises generating at least one of the items of personalized infonnation from infomiation 
on content available from a Web site (28, 29). 

7. A metiiod as in claim 1 wherein the selected item of personalized infonnation 
20 comprises a reference to a Web site, fiirther comprising: 

identifying a selection of tiie Web site reference by tiie viewer (50); and 
displaying content from tiie Web site on tiie viewing device (28, 29, 46). 

8. A metiiod as in claim 7 wherein tiie Web site comprises a push server (29) and 
25 a collaborative filtering database (26), tiie Web site displaymg step comprising: 

identifying differences (27) between tiie viewer profile (12) and similar user 

profiles in tiie collaborative database (26); and 

displaying content pushed from tiie push server on tiie viewing device in 

accordance witii tiie differences (29, 46). 

30 

9. A metiiod as in claim 8 wherein tiie Web site comprises a profile minor (27) 
and wherein tiie difference identifying step comprises: 

downloading tiie viewer profile from an adaptive viewer profile database (12) 

to the profile minor (27); 
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updating the profile mirror from the collaborative filtering database (26) to 
obtain an updated profile mirror (27); and 

pushing content from the push server (29) in accordance with the updated 
profile mirror for display (46) on the viewing device. 

5 

10. A method as in claim 9 further comprising uploading the updated profile 

mirror from the profile mirror (27) to the adaptive viewer profile database (12). 



11. A receiver apparatus for obtaining content from multiple information sources 
10 (20) for viewing (40) by a viewer, comprising: 

a program detector (22); 

an program-related additional information detector (24); 

a video output coupled to the program detector for driving a viewing device 
(40) to present the program (42) to the viewer; 
15 an adaptive viewer profile database (12); 

an implicit filter (14) coupled to the adaptive viewer profile database, the 
implicit filter having an input coupled to the program-related additional information detector 
for generating items of personalized information based on the program-related additional 
information and on the adaptive viewer profile database; 
20 means (44) for superimposing a visual selection aid (5) over the program, the 

visual selection aid comprising index symbols (1, 2, 3) for the items of personalized 
information and the index symbols on the visual selection aid being accessible to the viewer 
for selection; 

means (50) for identifying a selection of one of the index symbols by the 
25 viewer to obtain a selected one of the items of personal information; and 

means (44, 46) for superimposing the selected item of personalized 
information over the program. 

12. A receiver apparatus as in claim 1 1 further comprising means (14) for 

30 updating the adaptive viewer profile database in accordance with the viewer's selection of the 
selected item of personalized information. 

13. A receiver apparatus as in claim 12 wherein the implicit filter further 
comprises: 
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means 



10 



15 



(27, 14) for generating at least one of the items of personalized 
information based on user profiles in a remote collaborative database (26) similar to the 
viewer profile, at least one of tiie index symbols on die visual selection aid being an 

additional index symbol therefor; and 

means (14, 50) for updating die viewer profile in accordance with a viewer's 
selection of the item of personalized information based on the similar user profiles in the 
collaborative database. 

14 A receiver apparattis as in claim 1 1 further comprising means (27, 14) for 
augmenting the adaptive viewer profile database in accordance with a collaborative data base 
(26). 

15 A computer program product comprising a computer readable medium having 
program logic recorded thereon for enabling a computer-enabled apparatus to display content 
from multiple information sources to a viewer on a viewing device, comprising: 

means (1 00, 110, 112) for establishing a viewer profile for the viewer based on 

various interests of the viewer; 

means for presenting a program (22) to the viewer on the viewing device (40, 

42); 

means (1 14, 124. 144) for generating items of personalized information based 

on the program and on the viewer profile; 

means for superimposing a visual selection aid (5) over the program, the visual 
selection aid comprising index symbols (1. 2, 3) for the items of personalized information, 
the index symbols on the visual selection aid bemg accessible to the viewer for selection; 

means (1 50) for identifying a selection of one of the index symbols by the 
viewer to obtain a selected one of the items of personal information; and 

means for superimposing tiie selected item of personalized information (44, 

46) over the program. 

30 16 A computer program product as in claim 1 5 fiirther comprising means (152) 

for updating the viewer profile in accordance witi. the viewer's selection of the selected item 
of personalized information. 



20 



25 
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1 7. A computer program product as in claim 1 5 wherein the means for generating 
personalized information comprises: 

means for generating at least one of the items of personalized information (27, 
29) based on user profiles in the collaborative database (26) similar to the viewer profile (12), 
at least one of the index symbols on the visual selection aid being an additional index symbol 
therefor; 

means for identifying a selection (50) of the index symbol for the item of 
personalized information based on the similar user profiles in the collaborative database; 

means for updating the viewer profile (12, 14) in accordance with the viewer's 
selection of the item of personalized information based on the similar user profiles in the 
collaborative database; and 

means for superimposing the item of personalized information (46) based on 
the similar user profiles in the collaborative database over the program. 

18. A computer program product as in claim 1 5 fiirther comprising means for 
augmenting the viewer profile in accordance with a collaborative data base (26). 
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